Skip to content

Surface contextual bandits, and match trackingCallback params to the SDK version - #113

Merged
gazzdingo merged 34 commits into
mainfrom
worktree-fix-tracking-callback-warning
Sep 2, 2026
Merged

Surface contextual bandits, and match trackingCallback params to the SDK version#113
gazzdingo merged 34 commits into
mainfrom
worktree-fix-tracking-callback-warning

Conversation

@gazzdingo

@gazzdingo gazzdingo commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Two related pieces of work, both needing SDK 1.7.0 (bumped here).

DevTools flagged any trackingCallback that didn't take exactly 2 params as
broken. 1.7.0 passes a third userContext arg, so the expected arity now
depends on the page's SDK version. Contextual bandits, new in 1.7.0, were
invisible in DevTools entirely: their rules carry variations under
contextualVariations so pre-1.7 SDKs skip them, and the Experiments tab only
looked at variations.

Changes

Tracking callback

  • Expect 3 params on SDK 1.7.0+ and 2 below it, rather than always 2. Both
    wrong-arity cases read as issues, with a panel message naming the version.
  • Stay quiet when the param list can't be determined, instead of warning.
  • Show the detected param count on the SDK tab row (Found (3 params)).
  • Replace the /\(([^)]+)\)/ param scrape, which truncated on destructured
    params and string defaults and misread single-param arrow functions.
  • Forward the third arg through the patched trackingCallback, and keep it on
    deferred tracking calls — fireDeferredTrackingCalls replays them with
    call.user.

Contextual bandits

  • Read contextualVariations ?? variations in getFeatureExperiments, so
    bandits appear in the Experiments tab at all.
  • Detect via contextualVariations; contextualBanditRef and the payload's
    contextualBandits block only appear once a bandit has trained contexts.
  • Contextual Bandit badge on the experiment row, detail header, and matching
    Event Logs entries.
  • Detail section with the current leaf, variation weights as percentages, the
    attributes the matched context tested, and setup checks. Weights and context
    collapse, since a bandit can have many of either.
  • Distinguish the states that otherwise look alike: no trained contexts yet,
    definition missing from the payload, no matching context, not bucketed, and
    a forced variation or missing hash attribute suppressing the weights.
  • Evaluate bandit experiments from the feature result rather than run(),
    which skips the feature-rule path and so reported a variation the page
    wasn't serving.
  • Hide Targeting and Traffic for bandits, whose static rule weights contradict
    the dynamic ones.

Fixes found along the way

  • useGBSandboxEval stuffed a single-element meta array into rules that had
    none, however many variations they had. The SDK indexes experiment.meta by
    variation, so getExperimentResult threw, the whole evaluation died, and
    every experiment rendered as Inactive with a null value. Pre-existing and
    unrelated to bandits — one meta-less rule anywhere took down the tab.
  • Use a dropdown for variation selection past four variations, via the app's
    SelectField so the menu portals correctly inside the detail panel.
  • Stop echoing In experiment and Force via dev tools in the debug box, and
    hide it when empty.

Testing

  • yarn test — 52 unit tests covering param parsing, version-dependent arity,
    bandit detection, and the stuffed-meta length. realPayload.test.ts pins the
    rule shape a live payload actually serves.
  • Load dist/ unpacked against a site on SDK 1.7.0. SDK tab → Tracking
    Callback shows the param count; a 2-param callback reads as an issue.
  • A contextual bandit shows the badge in the Experiments list and a Contextual
    Bandit section in the detail. With no trained contexts in the payload, expect
    the rule's variation weights and no context section.

Dependencies

None.
image
image
image

The SDK types allow both `(experiment, result)` and
`(experiment, result, userContext)` tracking callbacks, but DevTools
flagged anything other than exactly 2 params as an implementation
problem. Treat 2 or 3 params as valid, and stay quiet instead of
warning when the param list can't be determined at all.

Also:
- Share the logic between the SDK tab, the panel copy, and the
  background icon status so they can't drift.
- Replace the `/\(([^)]+)\)/` param scrape, which truncated at
  destructured params and default values containing parens, and
  mis-parsed single-param arrow functions.
- Forward extra args through the patched trackingCallback and
  onFeatureUsage wrappers so a userContext arg isn't swallowed.
DevTools derives its "latest SDK version" from the bundled SDK's
package.json, so a stale dependency means stale outdated-version
warnings. 1.7.0 also passes a userContext arg to instance-level
trackingCallbacks, which the arg forwarding in the previous commit
now preserves.
The row now reads "Found (3 params)" and the panel names the detected
signature, so the callback shape is visible without opening the page's
source.
Self-contained page for exercising the SDK tab with no GrowthBook
account: apiHost points at the same static server, and a static
api/features/local-test payload satisfies the canConnect probe.
?params=1|2|3|4 switches the trackingCallback arity and &ofu=1 supplies
an onFeatureUsage callback.

The SDK bundle is copied in by `yarn test-page` rather than vendored.
onFeatureUsage only ever lands in the SDK's user context, which core.ts
calls as cb(key, result); the 3-arg form is GrowthBookClient's global
context, which DevTools never patches. So the rest param was always
empty and the cast only silenced a type error for an argument that
cannot arrive.

trackingCallback genuinely does receive a third userContext arg in 1.7+,
so that forwarding stays - but now via a typed optional param rather
than a rest spread plus a cast.
- deferred tracking calls kept only {experiment, result}, but
  fireDeferredTrackingCalls replays them as
  trackingCallback(experiment, result, call.user), so a replayed
  3-param callback got user === undefined
- the wrapper discarded the callback's return value, so the SDK could
  no longer await an async trackingCallback
- a zero-arity forwarding wrapper parsed to [] and warned; that shape is
  common in minified analytics code and works fine
- parseCallbackParams desynced on brackets and commas inside string
  defaults, eg (experiment, result, sep = ",")
- the toString guard could throw a TypeError out of the injected script
  when a page shadowed toString to return a non-string
- the row read "Found (1 params)" and counted rest params, disagreeing
  with the detail panel

Also makes the bare-arrow test exercise the branch it names - a type
annotation forced parens, so deleting the branch left the suite green.
- parseCallbackParams no longer wraps toString in try/catch with a
  typeof guard; main never did and callers pass a real function
- trim multi-line comments down to one line each
- flatten the nested ternary behind the SDK tab row label
- straighten out the trackingCallback panel branches so each renders a
  whole sentence instead of splicing a conditional before the period
- drop the any casts from the tests
1.7.0 is where the SDK started calling the instance-level callback as
(experiment, result, userContext); through 1.6.5 it only ever passed
two args. So the expected arity is exact in both directions - two params
on 1.7.0+ silently drops the userContext payload, and three on anything
older leaves that param permanently undefined. Both now read as issues,
with a panel message naming the version and what is lost.

Stays lenient when the page's SDK version is unknown.
The third param is not what carries the user's attributes - omitting it
just means missing newer features like contextual bandits.
Contextual bandit rules keep their variations under contextualVariations
so pre-1.7 SDKs skip them, but getFeatureExperiments gated on
rule.variations - so they never reached the Experiments tab at all.
Read either field, then mark and explain the ones that are bandits.

- "Contextual Bandit" badge on the experiment row, the detail header,
  and matching exposure entries in Event Logs
- detail section with the weights actually applied to this user, as
  percentages to match Rule.tsx, and the attributes the matched context
  tested - both collapsible, since a bandit can have many of either
- calls out the three states that otherwise look like a healthy bandit:
  ref missing from the payload, no matching context, and not bucketed

Weights come from the feature's experimentResult: the SDK applies bandit
logic while evaluating the rule, so gb.run() and the raw rule never
carry them.
A live payload marks a contextual bandit rule with contextualVariations
alone: contextualBanditRef and the top-level contextualBandits block only
appear once the bandit has trained contexts. Keying detection off the ref
meant a real bandit rendered as an ordinary experiment.

Detection now keys off contextualVariations, and the detail panel handles
a bandit with no trained contexts by showing the rule's fallback weights
and saying that is what every user gets.
It sat just above Implementation, well below the fold, so a bandit read
as an ordinary experiment on open. It now follows Current value, and its
heading matches the label style of the sections around it.
- lead with a meta box: bandit ref, current leaf, bandit version
- "Variation Weights" replaces the fallback wording, and the assigned
  variation is emphasised in the bar list
- hide Targeting and Traffic for bandits, whose static rule weights
  contradict the dynamic ones shown above
- render context values directly rather than through ValueField
- ref, current leaf and version as a compact sub-line under the heading
- leaf shown as a pill beside the weights label when a context matched
- Setup block with the mockup's checks: whether trackingCallback receives
  userContext (bandit rewards can't be attributed without it) and whether
  a bandit definition is in the payload
- weights read "Weights for this context" when a context matched, and
  "Variation Weights" otherwise
- drop the unused getContextualBandit export and un-export the types and
  condition walker that nothing outside the module names
- cut multi-line explanatory comments down to one line each
- use clsx for the conditional weight-row classes, as elsewhere in the app
- cover the condition walker through getMatchedContextAttributes now that
  it is private
@gazzdingo

Copy link
Copy Markdown
Contributor Author

@greptileai

Result.key is the variation key (meta.key, or the variation index), not
the experiment key, so matching it against experiment.key found nothing:
the panel always reported no trained contexts and fell back to the rule's
static weights. Match on the feature result's experiment key instead, and
only build the context object when bandit weights are actually present.
- evaluate bandit experiments from the feature result rather than run(),
  which skips the feature-rule path and so reports a variation the page
  is not serving
- read that result from the parent instead of mounting a third sandbox
  evaluation in the detail panel
- detect bandits in the logs via the runtime experiment's contextualBandit;
  the ref and contextualVariations are stripped by the time a log is built
- reuse ContextualBanditBadge in the experiments list and logs
- stop reporting an untrained bandit as a warning, and stop blaming the
  callback for a userContext an older SDK never passes
- say "Not applied for this user" when a definition exists but no bandit
  weights were used, rather than claiming the payload has none
For a bandit with no trained contexts the sub-line read "leaf: No trained
contexts yet", labelling a leaf that does not exist and repeating what the
setup check says. The leaf now only appears once weights were applied, and
a definition that exists but was not used is called out in the checks.
Radio cards stop being scannable once an experiment has more than a
handful of variations, which a bandit routinely does.

Also say what to do when a forced variation is hiding the bandit weights:
forcing sets hashUsed false, and the SDK then drops leafId and
variationWeights entirely.
The dropdown used position="popper", unlike every other Select in the
app; inside the fixed, scrolling detail panel its menu did not open.
Match the existing variant="soft" usage.

The leaf was buried in the grey sub-line and hard to find, so it now has
its own labelled row.
Without a value for the rule's hashAttribute the SDK logs "Skip because
missing hashAttribute" and returns no result at all, so the panel said
only that weights were not applied. It now names the attribute.
Selecting by value reads better than by index when the variations are
plain strings, as a bandit's usually are. Object-valued variations keep
the syntax-highlighted display, where a dropdown of JSON blobs would be
unreadable.
…the panel

Current value goes back to the read-only value display.

The variation dropdown rendered but would not open: the detail panel sets
z-index 1000 inline, and the portaled menu landed underneath it. Give the
menu content a higher z-index and keep popper positioning, which the nav
select already uses.
useGBSandboxEval stuffs rule metadata for its debug log, but when a rule
had no meta it substituted a single-element array however many variations
the rule had. The SDK indexes experiment.meta by variation index, so
getExperimentResult threw on any variation past the first, the whole
evaluate() call died, and every experiment then rendered as Inactive with
a null value - the bandit panel included.

Build the stuffed meta per variation, keeping any entries the rule already
had, and keep the single tag for rules with no variations since the debug
log reads meta[0].
Three attempts at making a Radix Select open inside the detail panel
failed. The app already has SelectField, which portals its menu through
SelectMenuPortalProvider - the component that exists precisely because
menus in these panels need an explicit portal target. Use it, keeping the
variation icons via formatOptionLabel and sort disabled so the options
stay in variation order.

Also hide the debug box under Enrollment Status when there is no message;
lastDebugLog defaults to an empty string, which rendered an empty console
panel.
…g lines

SelectField labels the selected value with the raw value, so the closed
dropdown read "1" rather than the variation name. Derive the name from the
index in formatOptionLabel, which covers the trigger and the menu.

Also stop echoing "In experiment" and "Force via dev tools" in the debug
box: the panel states both itself, right above it.
Both sections used a plain Text heading with the caret, which read as a
title rather than a toggle and sat tight against the row above. Use the
bold Link trigger and accordion my-4 spacing that SdkItemPanel already
uses for its collapsible sections.
Both accordion bodies started flush against the toggle.
Reverts the content margin from 180abd8: the gap belongs between the
previous section and the toggle, so each trigger now carries the top
margin.
@gazzdingo gazzdingo changed the title Accept 3-param tracking callbacks in SDK health check Surface contextual bandits, and match trackingCallback params to the SDK version Aug 31, 2026
- Rule.tsx destructured rule.variations, so a bandit rule rendered in the
  Features tab as a rollout with no variations. Use ruleVariations, the
  same fix already applied to getFeatureExperiments.
- Don't highlight variation 0 for a user the SDK skipped: it clamps the
  index to 0 with inExperiment false, so the weights list contradicted the
  Inactive status shown above it.
- The missing-hash-attribute diagnostic sat behind , which the
  common bandit shape never has, so it never fired. Hoist it, and count
  fallbackAttribute as satisfying the hash value.
- Omit isContextualBandit when false: useSearch matches
  JSON.stringify(item), so a literal false made every experiment log row
  match a search for "bandit".
- Reserve room for the badge in split view, where the types container is
  absolutely positioned and the name ran underneath it.
Bandit experiments had an empty Results log: `banditResult ?? run()`
short-circuited run(), which is what populates the debug log. Always run,
and prefer the bandit result afterwards.

Remove the amber forced-variation warning, gated on `isForced && cb`,
which can never both be true - forcing sets hashUsed false and the SDK
then drops the weights that produce cb. The setup check already covers
that case with something actionable.

Gate the panel on the page's SDK version. DevTools evaluates with its own
bundled SDK, so on a pre-1.7 page it would report a bandit assignment the
page never made; say so instead.
@gazzdingo
gazzdingo merged commit f1ffac8 into main Sep 2, 2026
1 check passed
@gazzdingo
gazzdingo deleted the worktree-fix-tracking-callback-warning branch September 2, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants